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Analysis  of  En  Route  Operational  Errors: 
Probability  of  Resolution  and  Time-On-Position 


On  March  14  2005,  the  Federal  Aviation  Administra¬ 
tion  (FAA)  AirTraffic  Organization  (ATO)  began  using  its 
Safety  Management  System  (SMS)  to  assess  all  changes  to 
the  National  Airspace  System  (NAS).  As  reflected  in  FAA 
Order  JO  1000.37,  the  ATO  SMS  process  is  designed  to 
identify,  assess,  mitigate,  and  document  the  acceptance 
of  acceptable  risk  associated  with  technological  and  pro¬ 
cedural  changes  to  the  NAS  (FAA,  2007).  Effective  use 
of  the  SMS  prevents  the  introduction  of  unacceptable 
safety  risk  into  the  NAS.  Of  course,  before  future  risks 
can  be  properly  evaluated,  it  is  first  necessary  to  assess 
the  current  risks  to  safety.  One  component  of  the  ATO’s 
SMS  is  its  safety  assurance  requirement,  which,  among 
other  things,  involves  the  monitoring  and  tracking  of 
safety-related  data. 

Historically,  the  ATO  has  used  the  rate  of  air  traffic 
control  (ATC)  operational  errors  (OEs)  as  one  of  its 
important  safety  metrics.  FAA  Order  7210-56C  defines 
an  OE  as:  “An  occurrence  attributable  to  an  element  of 
the  air  traffic  system  in  which: 

1.  Less  than  the  applicable  separation  minima  results 
between  two  or  more  aircraft,  or  between  an  air¬ 
craft  and  terrain  or  obstacles  (e.g.,  operations  below 
minimum  vectoring  altitude  (MYA);  equipment  / 
personnel  on  runways),  as  required  by  FAA  Order 
71 10.65  or  other  national  directive;  or 

2.  An  aircraft  lands  or  departs  on  a  runway  closed  to 
aircraft  operations  after  receiving  air  traffic  autho¬ 
rization;  or 

3.  An  aircraft  lands  or  departs  on  a  runway  closed  to 
aircraft  operations,  at  an  uncontrolled  airport  and  it 
was  determined  that  a  NOTAM  regarding  the  run¬ 
way  closure  was  not  issued  to  the  pilot  as  required 
(p.  5-1,  FAA,  2009a).” 

Although  tracking  OE  rates  over  time  is  one  way 
of  assessing  the  affect  of  technological  and  procedural 
changes  to  the  NAS,  we  argue  that  there  are  at  least  two 
other  metrics  that  can  enhance  the  understanding  of 
safety  risk:  (a)  OE  containment  (and  its  corresponding 
probability  of  resolution)  and  (b)  the  probability  of  an  OE 
occurring  based  on  how  long  controllers  are  on  position. 
The  remainder  of  this  report  describes  the  results  of  two 
studies  that  highlight  the  important  contributions  that 
both  of  these  metrics  make  to  the  SMS  process. 


STUDY  1:  OE  CONTAINMENT 

The  ATO  classifies  OEs  into  one  of  four  safety  risk 
categories  based  on  the  percentage  of  the  prescribed 
separation  minima  (called  separation  conformance) 
that  exists  between  the  two  aircraft  at  the  time  of  closest 
proximity.  Four  safety  risk  categories  are  used  to  describe 
the  relative  severity  of  an  OE: 

1 .  Proximity  Events:  An  OE  where  90%  or  greater  sepa¬ 
ration  is  retained  in  either  the  horizontal  or  vertical 
plane  and  does  not  include  any  violation  of  wake 
turbulence  separation  minima.  Although  proximity 
events  are  technically  OEs,  they  are  not  viewed  as  a 
threat  to  aviation  safety. 

2.  Category  C  (low  risk):  A  loss  of  airborne  separation 
where  the  separation  conformance  is  greater  than 
or  equal  to  75%,  but  the  horizontal  and  vertical 
separation  retained  is  less  than  90%  of  the  required 
separation.  In  events  with  wake  turbulence  violations 
the  lateral  separation  retained  is  greater  than  or  equal 
to  85%  but  less  than  100%. 

3.  Category  B  (moderate  risk):  A  loss  of  airborne  sepa¬ 
ration  where  the  separation  conformance  is  greater 
than  or  equal  to  34%  but  less  than  75%.  In  events 
with  wake  turbulence  violations  the  lateral  separa¬ 
tion  retained  is  greater  than  or  equal  to  70%,  but 
less  than  85%. 

4.  Category  A  (high  risk):  This  is  the  most  severe  type 
of  OE  and  involves  a  loss  of  airborne  separation 
where  the  separation  conformance  is  less  than  34%. 
In  events  with  wake  turbulence  violations,  the  lateral 
separation  retained  is  less  than  70%. 

OE  severity  classification  can  be  readily  made  using 
the  charts  provided  in  the  appendices  of  FAA  Order 
7210.56C  (FAA,  2009a).  In  our  study,  we  used  the  Non- 
Wake  Separation  Conformance  Categorization  chart  of 
Appendix  9  (shown  as  Figure  1)  and  two  Conformance 
Categorization  tables  from  Appendix  1 1  (shown  as  Tables 
1  and  2). 

Currently  the  ATO  tracks  the  number  of  Categories 
A&B  OEs  as  a  measure  of  safety.  The  number  of  Category 
A&B  OEs  is  divided  by  the  number  of  aircraft  operations 
expressed  in  millions.  This  rate  is  then  used  to  establish 
and  to  monitor  progress  toward  achieving  safety  goals. 
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Horizontal  Separation  Maintained 

Figure  1.  Non-Wake  Separation  Categorization  (from  Appendix  9, 
FAA  Order  721 0-56,  FAA  2009b). 


Table  1.  OE  Severity  categorization  with  5  nautical  miles  (nm)  lateral  separation  and 
1000-foot  vertical  separation  minima,  non-wake  condition 


Vertical 


Lateral 

0 

100 

200 

300 

400 

500 

600 

700 

800 

900 

feet 

ft 

ft 

ft 

ft 

ft 

ft 

ft 

ft 

ft 

4.99- 
4.50  NM 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

4.49- 
3.75  NM 

C 

C 

C 

C 

C 

C 

C 

C 

C 

PE 

3.74- 
3.72  NM 

B 

C 

C 

C 

C 

C 

C 

C 

C 

PE 

3.71- 
3.62  NM 

B 

B 

C 

C 

C 

C 

C 

C 

C 

PE 

3.61- 
3.44  NM 

B 

B 

B 

C 

C 

C 

C 

C 

C 

PE 

3.43- 
3.18  NM 

B 

B 

B 

B 

C 

C 

C 

C 

C 

PE 

3.17- 
2.80  NM 

B 

B 

B 

B 

B 

C 

C 

C 

c 

PE 

2.79- 
2.25  NM 

B 

B 

B 

B 

B 

B 

C 

C 

c 

PE 

2.24- 
1.70  NM 

B 

B 

B 

B 

B 

B 

B 

C 

c 

PE 

1.69- 
1.63  NM 

A 

B 

B 

B 

B 

B 

B 

C 

c 

PE 

1.62- 
1.38  NM 

A 

A 

B 

B 

B 

B 

B 

C 

c 

PE 

1.37- 
1.35  NM 

A 

A 

A 

B 

B 

B 

B 

C 

c 

PE 

1.34- 
0.80  NM 

A 

A 

A 

B 

B 

B 

B 

B 

c 

PE 

0.79- 
0.00  NM 

A 

A 

A 

A 

B 

B 

B 

B 

c 

PE 

From  Appendix  11,  FAA  Order  7210-56,  FAA  2009b. 
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Table  2.  OE  Severity  categorization  with  5  nautical  mile  (nm)  lateral  separation  and  2000-foot 
vertical  separation  minima,  non-wake  condition 


Vertical 


Lateral 

0 

feet 

100 

ft 

200 

ft 

300 

ft 

400 

ft 

500 

ft 

600 

ft 

700 

ft 

800 

ft 

900 

ft 

4.99- 
4.50  NM 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

4.49- 

3.75  NM 
3.74- 
3.72  NM 
3.71- 
3.68  NM 

C 

C 

C 

C 

C 

C 

C 

C 

C 

C 

B 

B 

C 

C 

C 

c: 

C 

C 

C 

C 

B 

B 

B 

C 

C 

c 

C 

C 

C 

C 

3.67- 

3.62  NM 

B 

B 

B 

B 

C 

c 

C 

C 

C 

C 

3.61- 
3.54  NM 

B 

B 

B 

B 

B 

c 

C 

C 

C 

C 

3.53- 
3.44  NM 
3.43- 
3.32  NM 

B 

B 

B 

B 

B 

B 

c 

C 

c 

C 

B 

B 

B 

B 

B 

B 

B 

C 

c 

C 

3.31- 
3.18  NM 

B 

B 

B 

B 

B 

B 

B 

B 

c 

C 

3.17- 

3.00  NM 

B 

B 

B 

B 

B 

B 

B 

B 

B 

c 

2.99- 

1.70  NM 

B 

B 

B 

B 

B 

B 

B 

B 

B 

B 

1.69 

NM 

1.68- 
1.63  NM 

A 

B 

B 

B 

B 

B 

B 

B 

B 

B 

A 

A 

B 

B 

B 

B 

B 

B 

B 

B 

1.62- 

1.53  NM 

A 

A 

A 

B 

B 

B 

B 

B 

B 

B 

1.52- 

1.38  NM 

A 

A 

A 

A 

B 

B 

B 

B 

B 

B 

1.37- 

1.16  NM 

A 

A 

A 

A 

A 

B 

B 

B 

B 

B 

1.15- 
0.80  NM 
0.79- 
0.00  NM 

A 

A 

A 

A 

A 

A 

B 

B 

B 

B 

A 

A 

A 

A 

A 

A 

A 

B 

B 

B 

Lateral 

1000 

feet 

1100 

Feet 

1200 

feet 

1300 

feet 

Yet 

1400 

feet 

tical 

1500 

feet 

1600 

feet 

1700 

feet 

1800 

feet 

1900 

feet 

4.99- 
4.50  NM 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

PE 

4.49- 
2.80  NM 

C 

C 

C 

C 

C 

C 

C 

C 

PE 

PE 

2.79- 
2.55  NM 
2.54- 
2.25  NM 
2.24- 
1.88  NM 

B 

C 

C 

C 

C 

C 

C 

C 

PE 

PE 

B 

B 

c 

C 

c 

c: 

C 

c 

PE 

PE 

B 

B 

B 

C 

c 

C 

C 

c 

PE 

PE 

1.87- 
1.35  NM 

B 

B 

B 

B 

c 

C 

C 

c 

PE 

PE 

1.34- 
0.00  NM 

B 

B 

B 

B 

B 

C 

C 

c 

PE 

PE 

From  Appendix  11,  FAA  Order  7210-56,  FAA  2009b. 
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As  of  this  writing,  the  FY2009  OE  goal  is  to  not  exceed 
2.10  Category  A&B  OEs/million  aircraft  operations 
(FAA,  2009b). 

The  emphasis  on  tracking  category  A&B  OEs  as  in¬ 
dicators  of  NAS  safety  is  consistent  with  the  philosophy 
of  tracking  errors  of  consequence,  rather  than  operator 
errors  in  general;  a  common  theme  that  emerged  at  the 
Fifth  Australian  Aviation  Psychology  Symposium,  Manly 
Beach,  Sydney  2000.  Representing  this  line  of  thinking, 
Maurino  (2003)  stated  that: 

“There  is  nothing  inherently  wrong  or  troublesome 
with  error  itself,  as  a  manifestation  of  human  behavior. 
The  trouble  with  error  in  aviation  lies  with  the  negative 
consequences  it  may  generate  in  operational  contexts. 
This  is  a  fundamental  point:  in  aviation  an  error  is  incon¬ 
sequential  if  the  negative  consequences  of  it  are  trapped 
before  it  produces  damage.  In  operational  contexts,  er¬ 
rors  that  are  caught  in  time  do  not  produce  damaging 
consequences  and  therefore,  for  practical  purposes,  do 
not  exist.  Countermeasures  to  error,  including  training 
interventions,  should  not  be  restricted  to  attempts  to 
avoid  errors,  but  rather  to  make  them  visible  and  trap 
them  before  they  produce  damaging  consequences  (pp. 
11-12).” 

Echoing  Maurino’s  remarks,  Toller  (2003)  challenged 
operators  to  think  of  error  management  as  consisting  of 
two  components:  (a)  error  prevention,  which  involves 
understanding  why  mistakes  are  made  and  taking  steps  to 
prevent  them,  and  (b)  error  containment,  which  involves 
recognizing  that  errors  are  inevitable  and  taking  steps  to 
avoid  the  possibility  of  single-point  failures. 


Progress  toward  error  prevention  goals  can  be  tracked 
through  existing  safety  metrics;  however,  progress  toward 
error  containment  goals  requires  additional  metrics,  such 
as  the  Probability  of  Resolution  (POR)  index  that  we 
propose.  The  POR  is  a  measure  of  the  efficiency  with 
which  the  NAS  is  able  to  resolve  separation  losses  (i.e., 
contain  the  errors  through  the  actions  of  controllers  and 
pilots)  before  they  degrade  into  greater  risks  to  safety. 

We  conceive  of  the  four  OE  safety  risk  categories  as 
representing  zones  through  which  point-in-time  separa¬ 
tion  losses  progress  until  they  reach  a  minimum.  Thus, 
each  of  the  OE  safety  risk  categories  represents  a  potential 
containment  field.  As  shown  in  Figure  2,  all  losses  ofsepa- 
ration  begin  with  a  point-in-time  separation  between  two 
aircraft  that  is  less  that  100%  of  separation  conformance. 
This  point-in-time  separation  degrades  until  a  situation  is 
reached  when  the  separation  between  two  aircraft  is  at  a 
minimum  and  no  further  degradation  occurs.  It  is  at  the 
minimum  that  the  amount  of  separation  conformance 
is  determined,  and  the  event  is  classified  into  one  of  the 
four  OE  safety  risk  categories. 

Using  Figure  2  as  a  conceptual  framework,  we  de¬ 
fine  the  POR  as  the  number  of  point-in-time  losses  of 
separation  resolved  (i.e.,  contained)  within  a  given  zone 
of  separation  conformance  divided  by  all  the  point-in 
time  losses  of  separation  that  were  eligible  to  be  resolved 
(that  is,  the  point-in-time  losses  of  separation  that 
passed  through  a  given  zone  of  separation  conformance) 
within  that  same  zone  of  separation  conformance.  Us¬ 
ing  the  OE  classification  system  amounts  to  calculating 
the  number  of  OEs  within  a  given  safety  risk  category 


Figure  2.  Operational  error  severity  defined  by  the  percentage  of  separation  conformance 
remaining. 
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divided  by  all  the  OEs  that  were  equal  to  or  more  severe 
than  the  OE  category  in  question.  This  is  shown  by  the 
following  equation: 


PORm 


(Eql) 


i  i=m 

where: 

POR=  the  Probability  of  Resolution  for  a  given  OE 
safety  risk  category  m 

OE  =  the  number  of  OEs  for  a  given  OE  safety  risk 
category  m 

m  =  a  given  OE  safety  risk  category  in  numerical  order 
1-4  corresponding,  respectively,  to  Proximity  Events, 
Category  C  OEs,  Category  B  OEs,  Category  A  OEs. 


Notice  that,  as  we  calculate  the  POR  for  higher  OE 
categories,  we  eliminate  from  the  denominator  the 
number  of  OEs  from  the  lower  risk  categories.  This  is 
called  computing  probabilities  without  replacement.  This 
means  that  previous  events  are  removed  (not  replaced) 
on  subsequent  sampling.  For  example,  the  POR  formula 
for  category  C  OEs  is  as  follows: 


PORcatc  — 


OE 


CatC 


OPcatC  +  OPcatB  +  OPcatA 


(Eq.2) 


All  point- in-time  losses  ofseparation  that  were  resolved 
within  the  region  of  proximity  events  were  contained  and, 
thus,  were  not  eligible  to  be  classified  as  category  C  or 
higher  OE  severity.  Because  of  their  ineligibility,  those 
losses  ofseparation  were  not  included  in  the  denominator 
while  calculating  the  POR  for  the  region  of  separation 
conformance  defined  by  category  C  OEs. 

In  this  study,  we  conceive  of  the  OE  severity  categories 
as  representing  a  series  of  OE  containment  fields  com¬ 
plete  with  the  various  technologies  and  human  support 
that  are  used  to  detect  and  resolve  losses  of  separation. 
We  calculated  the  POR  for  each  OE  severity  category 
as  a  measure  of  the  efficiency  with  which  the  point-in¬ 
time  losses  of  separation  are  resolved  before  degrading 
to  separation  losses  associated  with  higher  safety  risks. 


We  then  demonstrate  how  the  POR  provides  additional 
information  that  should  be  included  as  a  safety  metric 
for  the  ATO’s  SMS. 


METHOD 


Data 

A  total  of  1293  OEs  were  extracted  from  an  en  route 
OE  research  data  base  for  the  period  May  1,  2001  to 
May  31,  2003.  The  extraction  included  the  following 
variables:  OE  report  number,  the  lateral  and  vertical 
distances  recorded  at  the  time  of  closest  proximity,  and 
the  required  lateral  and  vertical  separation  standards. 
Because  an  OE  can  be  attributed  to  more  than  one 
controller,  we  extracted  data  for  only  the  controller  who 
was  primarily  responsible  for  the  OE.  This  was  to  avoid 
having  multiple  cases  associated  with  the  same  OE.  We 
accomplished  this  by  using  a  variable  which  was  flagged 
to  indicate  the  record  of  the  primary  controller.  Thus, 
no  specific  individual  was  matched  with  a  given  OE. 

The  dataset  chosen  for  this  study  was  the  same  one 
used  on  a  previous  proj  ect,  which  evaluated  the  validity  of 
the  operational  error  severity  index  used  at  that  time  and 
reported  in  Bailey,  Schroeder,  &  Pounds  (2005).  Because 
the  dataset  had  previously  been  used,  we  understood  its 
properties  particularly  with  regard  to  missing  data  fields. 
OE  reporting  underwent  a  number  of  changes  since 
2002,  each  of  which  produced  a  different  set  of  reporting 
procedures.  This  made  it  difficult  to  determine  whether 
data  were  missing  because  of  data  entry  errors  or  changes 
in  procedures.  Since  our  primary  goal  in  this  study  was 
to  demonstrate  the  utility  of  employing  a  measure  of  OE 
containment  for  SMS  purposes,  we  were  not  as  concerned 
about  using  current  OE  data  as  we  were  with  having  a 
dataset  that  we  understood. 

RESULTS 

Table  3  shows  the  number  of  OEs  and  corresponding 
PORs  associated  with  each  of  the  OE  severity  categories. 


Table  3.  Probability  of  Resolution 


OE  Severity 

OEm 

IMS 

§ 

PORm 

Proximity  Event 

333 

1293 

0.26 

Category  C 

719 

960 

0.75 

Category  B 

226 

241 

0.94 

Category  A 

15 

15 

1.00 

m  =  a  given  OE  severity  category 
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These  data  are  presented  graphically  in  Figures  3  and  4, 
respectively.  When  considering  the  en  route  centers  as 
an  aggregate,  we  see  that  the  NAS  was  26%  (333/1293) 
effective  at  resolving  losses  of  separation  within  the  prox¬ 
imity  event  range,  75%  (719/960)  effective  at  the  low 
risk  range  (Category  C),  94%  (226/241)  effective  at  the 
moderate  risk  range  (Category  B),  and  100%  (15/15) 
effective  at  the  high  risk  range  (Category  A).  Notice 


that,  in  Figure  3,  the  number  of  OEs  increased  going 
from  the  Proximity  Events  to  the  Low- Risk  Category  C 
events,  and  then  the  number  dropped  off  sharply  as  we 
progressed  through  Categories  B  and  A.  In  contrast  to 
Figure  3,  the  distribution  of  PORs  in  Figure  4  shows  a 
continuous  rise  in  efficiency  as  we  progressed  through  the 
OE  severity  categories  and  ended  with  1 00%  resolution 
by  the  time  we  reached  the  Category  A  region. 


OE  Severity 


Figure  3.  Distribution  of  OEs  by  OE  Severity. 


Figure  4.  Probability  of  Resolution  by  OE  Severity  Category. 
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Table  4.  Number  of  OEs  associated  with  a  given  category  of  OE  severity 


OE  Severity 

Separation 

Conformance 

Interval  Number  OEs 

Proximity  Event 

90%<X<100% 

11% 

333 

Category  C 

75%<X<90% 

15% 

719 

Category  B 

34%<X<75% 

40% 

226 

Category  A 

X<34% 

34% 

15 

Table  5.  Number  of  OEs  associated  with  each  10%  loss  of  separation  conformance 

Separation 

Interval 

Separation 

Conformance 

Interval 

Number  OEs 

POR 

1 

90%<X<100% 

10% 

333 

0.26 

2 

80%<X<90% 

10% 

540 

0.56 

3 

70%<X<80% 

10% 

252 

0.60 

4 

60%<X<70% 

10% 

84 

0.50 

5 

50%<X<60% 

10% 

41 

0.49 

6 

40%<X<50% 

10% 

19 

0.44 

7 

30%<X<40% 

10% 

18 

0.75 

8 

20%<X<30% 

10% 

6 

1.00 

9 

10%<X<20% 

10% 

0 

NA 

10 

0%<X<10% 

10% 

0 

NA 

Because  the  percentage  of  required  airspace  covered 
by  the  region  of  a  given  OE  category  did  not  represent 
equal  intervals  (see  Table  4),  we  wondered  if  the  trend 
of  increasing  POR  efficiency  would  be  replicated  using 
equal  separation  conformance  intervals.  That  is,  we  were 
curious  about  whether  certain  regions  of  inefficiency  might 
become  evident  if  equal  separation  conformance  intervals 
were  examined.  Thus,  we  eliminated  the  OE  severity 
categories  (Figure  1)  and  instead  divided  the  region  of 
separation  conformance  into  10  equal  percentage  inter¬ 
vals.  We  then  computed  the  number  of  OEs  associated 
with  each  interval  and  calculated  the  corresponding  PORs. 

The  number  of  OEs  and  corresponding  POR  for  each 
separation  conformance  interval  is  presented  in  Table  5 
and  graphically  represented  in  Figures  5  and  6,  respectively. 

Figure  5  shows  the  distribution  of  OEs  across  the  re¬ 
gions  ofseparation  conformance.  Notice  that  the  pattern  is 
similar  to  the  one  presented  in  Figure  3.  In  Figure  5,  there 


was  a  sharp  increase  in  the  number  of  OEs  going  from 
the  first  to  the  second  separation  conformance  interval, 
which  corresponds  to  the  transition  between  Proximity 
Events  and  Low  Severity  OEs  shown  in  Figure  3.  Follow¬ 
ing  this  increase  in  the  number  of  OEs,  in  both  figures 
there  is  a  continuous  decline  throughout  the  remainder 
of  the  separation  conformance  regions. 

In  contrast  to  the  similarities  in  the  OE  distributions, 
the  corresponding  POR  distributions  were  different. 
Whereas  in  Figure  4  we  saw  a  continuous  rise  in  efficiency 
of  OE  containment  as  we  progressed  through  the  OE  se¬ 
verity  categories,  in  Figure  6,  we  see  a  drop  in  efficiency  of 
OE  containment  occurring  between  the  third  and  fourth 
interval.  A  zone  of  relatively  lower  OE  containment  ef¬ 
ficiency  (intervals  4-6)  continues  until  reversing  at  the 
transition  between  the  sixth  and  seventh  interval.  This 
zone  is  primarily  in  the  Moderate  OE  severity  (Category 
B)  region  depicted  in  Figure  4. 
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Figure  6.  Probability  of  Resolution  for  each  Separation  Conformance  Interval. 


Table  6.  Operational  Error  Management 


Error  Prevention 

Error  Containment 

OE  Severity 

Number 

Percent  of  Total 

Number 

POR* 

Proximity 

333 

26% 

1293 

0.26 

Low 

719 

56% 

960 

0.75 

Moderate 

226 

17% 

241 

0.94 

High 

15 

1% 

15 

1.00 

*POR  =  Probability  of  Resolution 


With  the  previous  analyses  offering  a  cautionary  note, 
we  returned  to  our  original  purpose  and  reformatted  the 
information  in  Table  3  into  a  two-component  table  of  error 
management,  as  shown  in  Table  6.  Under  the  heading  of 
Error  Prevention,  we  show  the  number  and  percentage 
of  OEs  associated  with  a  given  category  of  OE  severity. 
For  Error  Containment,  we  show  the  number  of  losses 
of  separation  that  were  eligible  to  be  resolved  within  a 
given  OE  severity  category  and  the  corresponding  POR. 

DISCUSSION 

When  the  POR  is  included  as  a  safety  metric,  all  the 
necessary  ingredients  are  in  place  for  a  two-component 
Operational  Error  Management  System.  Looking  just 
at  the  error  prevention  indicators,  we  see  the  number  of 
separation  losses  that  were  resolved  within  each  of  the 
four  OE  severity  categories.  This  information  tells  us  how 
well  the  ATO  is  at  preventing  or  reducing  the  number 
and  severity  of  OEs.  This  is  the  kind  of  information  that 
one  needs  when  establishing  safety  goals.  However,  there 
is  a  risk  associated  with  using  just  these  kinds  of  num¬ 
bers.  The  prevention  numbers  in  and  of  themselves  do 
not  help  us  understand  how  well  controllers’  actions  and 
the  pilots’  responses  prevented  (i.e.,  contained)  an  initial 
loss  of  separation  from  getting  worse.  This  is  what  the 
POR  captures;  thus,  we  recommend  that  it  be  included 
as  an  additional  metric  for  the  ATO ’s  Safety  Management 
System.  However,  from  a  methodological  perspective, 
there  is  an  unresolved  issue  concerning  the  separation 
conformance  intervals  that  needs  to  be  addressed  before 
the  POR  is  used. 

In  the  results  section,  we  explained  how  the  lack  of 
equal  separation  conformance  intervals  masked  a  zone  of 
relatively  low  OE  containment  within  the  Moderate  OE 
severity  region.  In  fact,  within  this  region  the  efficiency 
of  OE  containment  continued  to  decline  before  taking 
a  dramatic  increase  at  the  last  time  interval  (interval  7). 
At  this  time,  one  can  only  speculate  as  to  the  cause  of 
this  phenomenon.  Perhaps  OEs  within  this  region  were 


surprises.  That  is,  the  controller  may  have  been  unaware 
that  an  OE  was  occurring  until  the  separation  loss 
crossed  the  75%  separation  conformance  threshold.  By 
the  time  the  OE  was  discovered,  the  controller  did  not 
have  sufficient  time  to  restore  separation  before  incurring 
a  further  loss. 

Regardless  of  the  reason  for  the  above  phenomenon,  it 
is  important  for  an  SMS  to  collect  data  at  the  finest  level  of 
detail  necessary  to  make  informed  decisions.  While  there 
may  be  valid  reasons  for  defining  the  official  categories  of 
OE  severity  as  they  are,  the  advantages  of  doing  so  must  be 
weighed  against  the  possible  obstruction  of  more  detailed 
information.  Methodologically  speaking,  equal  interval 
measurements  are  preferred  over  categorical  assignments 
and,  thus,  it  may  be  advantageous  for  the  ATO  to  adopt 
equal  separation  conformance  intervals  both  for  metrics 
of  OE  prevention  and  containment. 

Another  issue  that  was  raised  in  our  study  concerns 
the  region  of  airspace  defined  by  Proximity  Events.  This 
region  had  the  lowest  efficiency  rating  for  OE  contain¬ 
ment  even  when  equal  OE  intervals  were  used.  Some 
may  argue  that  with  sufficient  technology,  losses  of 
separation  should  be  predicted  well  in  advance  of  their 
occurrence  and,  thus,  should  be  readily  contained  within 
the  Proximity  Event  region.  One  of  the  current  advance 
warning  systems  is  the  conflict  probe  that  is  part  of  the 
User  Request  Evaluation  Tool  (URET),  installed  at  en 
route  centers  (Brudnicki  &MacFarland,  1997).  URET’s 
conflict  probe  software  attempts  to  model  the  future  state 
of  a  sector  by  using  the  current  flight  plan  of  aircraft  and 
projecting  their  trajectories  1 0-20  minutes  into  the  future. 
Given  the  dynamic  nature  of  ATC,  it  is  not  surprising  to 
find  that  URET’s  conflict  probe  produces  a  number  of 
false  alarms,  making  it  problematic  as  an  advance  warn¬ 
ing  system.  Thus,  in  practice,  URET  is  more  useful  as 
strategic  predictor  of  potential  aircraft  conflicts  rather 
than  an  actual  warning  of  an  impending  conflict.  The 
FAA  is  currently  developing  an  en  route  automation 
modernization  (ERAM)  system  with  an  improved  conflict 
detection  system  (Allendoerfer,  Willems,  Zingale,  &  Pai, 
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2006) .  The  improved  system  will  be  basically  like  URET, 
only  better  integrated  into  the  operational  software,  thus 
more  sensitive  to  the  dynamic  changes  of  ATC. 

Despite  the  desire  to  provide  conflict  warnings  to 
alert  controllers  of  potential  losses  of  separation,  it  is 
ultimately  the  responsibility  of  each  controller  to  keep 
aircraft  properly  separated  according  to  the  prescribed 
separation  minima.  Among  other  things,  this  requires  that 
controllers  use  their  professional  judgment  to  maintain 
a  safe  and  expeditious  flow  of  traffic  (FAA,  2009a).  It  is 
this  professional  judgment  that  comes  into  play  when¬ 
ever  a  controller  detects  that  he  or  she  has  allowed  an 
OE  to  occur.  Depending  on  the  circumstances,  once  a 
controller  has  discovered  that  an  OE  has  occurred,  he  or 
she  should  use  professional  judgment  to  restore  aircraft 
separation.  Consequently,  it  is  possible  that  a  controller 
may  choose  a  set  of  conflict  resolution  maneuvers  (via 
speed,  altitude,  and  heading  adjustments)  that  allow  the 
separation  minima  to  be  further  compromised  temporar¬ 
ily  to  prevent  a  disruption  in  the  existing  traffic  flow  or 
prevent  involved  aircraft  from  having  to  execute  maneuvers 
that  would  alarm  or  injure  the  passengers.  Of  course,  the 
greater  the  separation  losses  at  the  time  of  OE  detection, 
the  fewer  degrees  of  freedom  are  available  for  controllers 
to  execute  actions  based  on  their  professional  judgments. 
Nevertheless,  it  is  likely  that  losses  of  separation  involving 
90%  or  more  of  separation  conformance  will  remain  a 
region  in  which  controllers  may  choose  to  allow  greater 
losses  of  separation  to  occur  in  a  controlled  environment 
to  minimize  the  need  for  pilots  to  make  radical  changes  in 
their  flight  paths  while  aircraft  separation  is  being  restored. 

As  a  final  thought,  it  should  be  noted  the  metrics  used 
for  error  prevention  and  error  containment  are  system 
measures  and,  thus,  are  appropriate  for  use  in  monitoring 
the  safety  of  the  NAS.  Elowever,  the  measures  in  and  of 
themselves  provide  little  insight  about  the  human  factors 
associated  with  error  detection  and  error  resolution.  At 
this  level  of  analysis,  human  error  taxonomies  are  needed 
for  classifying  both  the  human  factors  associated  with  the 
causes  of  OEs  and  their  resolution.  A  number  of  classifica¬ 
tion  systems  are  suitable  for  analyzing  OE  human  factors 
causes.  These  include  the  Human  Factors  Analysis  and 
Classification  System,  HFACS  (Shappell  &  Weigmann. 
2000;  Scarborough,  Bailey,  &  Pounds,  2005),  JANUS 
(Pounds,  &  Isaac:  (2003),  and  the  Normal  Operations 
Safety  Survey,  NOSS  (Henry  et  al.,  2008).  None  of  the 
taxonomies,  however,  identify  the  types  of  judgment 
and  decision-making  strategies  that  are  employed  by 
controllers  when  resolving  OEs.  Thus,  it  remains  for 
future  research  to  determine  whether  existing  taxonomies 
can  be  adapted  to  capture  error  resolution  strategies  or 
whether  a  completely  different  set  of  ATC  judgment  and 
decision-making  measurements  should  be  developed.  In 


the  study  that  follows,  we  return  to  the  concept  of  error 
prevention  by  examining  the  impact  that  time  on  posi¬ 
tion  has  on  the  probability  of  an  OE  occurring.  Time 
on  position  refers  to  the  amount  of  time  a  controller  was 
actively  controlling  aircraft  before  experiencing  an  OE. 

STUDY  2:  TIME  ON  POSITION 

For  an  SMS  to  be  effective,  it  must  be  able  to  identify 
anomalies  or  departures  from  normal  operations.  Al¬ 
though  OEs  are  clearly  departures  from  normal  operations, 
there  often  exists  insufficient  information  about  what  is 
“normal.”  Instead,  it  is  assumed  that  for  OEs  to  occur, 
the  controller(s)  involved  must  have  done  something  dif¬ 
ferent  (i.e.,  non-normal);  otherwise,  OEs  would  be  more 
frequent  than  they  are.  Although  this  line  of  reasoning 
has  merit,  it  also  creates  a  trap  for  error  management. 
The  trap  is  associated  with  jumping  to  a  solution  before 
properly  understanding  the  problem. 

Consider  the  ubiquitous  finding  that  a  higher  percent¬ 
age  of  OEs  occur  early  on  position  and  then  taper  off 
as  the  time  on  position  (TOP)  increases.  This  trend  has 
been  consistently  observed  across  the  tower,  Terminal 
Radar  Approach  Control  (TRACON),  and  en  route 
options  and  times  of  day  (Schroeder,  Bailey,  Pounds,  & 
Manning,  2006).  The  trend  is  counterintuitive,  given 
what  is  known  about  time  on  task  and  mental  fatigue,  in 
which  lapses  of  attention  become  more  likely  as  time-on- 
task  increases  (Trejo  et  al.,  2007).  From  the  time  on  task 
perspective,  one  would  expect  that  the  longer  a  control¬ 
ler  is  on  position,  the  greater  the  chances  that  mental 
lapses  in  attention  would  occur.  However,  since  a  greater 
number  of  OEs  occur  earlier  on  position,  as  opposed  to 
later,  the  assumption  is  made  that  controllers  coming  on 
position  must  not  be  fully  prepared  to  manage  the  traffic 
situation  due,  in  part,  to  a  faulty  position  relief  briefing. 
Although  there  is  no  doubt  that  managing  the  transfer 
of  position  process  is  safety-critical,  it  has  been  difficult 
to  empirically  link  OEs  that  occur  early  on  position  with 
the  transfer  of  position  process. 

In  their  study  of  en  route  OEs  that  occur  early  on  posi¬ 
tion,  Bailey,  Pounds,  &  Scarborough  (2008)  found  that 
the  position  relief  briefing  process  was  seldom  reported 
as  a  contributing  factor  for  OEs  that  occurred  during  the 
first  1 0  minutes  following  a  position  transfer.  Although 
Bailey  and  his  colleagues  reported  problems  associated 
with  the  quality  of  data  contained  in  the  FAA’s  OE  da¬ 
tabase,  they  also  raised  the  possibility  that  the  early  on 
position  OE  may  be  the  result  of  an  exposure  effect.  They 
conjectured  that  a  greater  number  of  controllers  in  the 
NAS  were  on  position  for  at  least  a  1 0-minute  duration 
compared  to  any  other  time  interval.  That  is,  there  were 
more  opportunities  (i.e.,  greater  exposure)  for  OEs  to 
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occur  earlier  on  position  as  opposed  to  later.  However,  at 
the  time  of  their  report,  they,  like  other  researchers  (e.g. 
Lowry,  MacWilliams,  Still,  &  Walker,  2005),  had  been 
unable  to  obtain  sufficient  normal  operations  data  to  test 
the  hypothesis  of  an  exposure  effect.  Specifically,  due  to 
the  labor-management  agreements  in  place,  TOP  data 
describing  operations  during  times  when  OEs  were  not 
occurring  could  not  be  released,  even  for  research  purposes. 

Recently,  we  were  able  to  obtain  a  limited  amount  of 
data  describing  the  length  of  time  controllers  were  on 
position  when  OEs  were  not  occurring.  We  were  able  to 
match  these  data  with  the  length  of  TOP  at  the  onset  of 
an  OE.  Together,  the  two  datasets  allowed  us  to  calculate 
the  probability  that  an  OE  would  occur,  based  on  the 
length  of  time  a  controller  was  on  position,  henceforth 
referred  to  as  Time  on  Position  Probability  (TOPP).  We 
then  used  the  TOPP  to  determine  whether  an  exposure 
effect  was  influencing  the  TOP  distribution  of  OEs. 

METHOD 


Sample 

TOP  data  were  available  only  for  calendar  year  2006. 
Because  the  total  number  of  records  exceeded  the  capac¬ 
ity  of  some  of  the  software  used  for  our  analyses,  we 
restricted  the  sample  to  include  only  data  from  the  six 
en  route  facilities  having  the  highest  number  of  OEs  in 
2006  (hereafter  called  the  “Big  6”).  This  decision  was 
made  to  ensure  that  we  had  an  adequate  number  of  OEs 
to  include  in  the  numerator  of  our  probability  calcula¬ 
tions,  while  at  the  same  time  not  exceeding  the  limits 
of  our  software.  The  Big  6  facilities  were  the  Chicago, 
Washington,  Indianapolis,  New  York,  Cleveland,  and 
Atlanta  air  route  traffic  control  centers  (ARTCCs).  Be¬ 
cause  the  Big  6  also  had  the  highest  number  of  aircraft 
operations,  their  individual  OE  rates  were  sometimes 
lower  than  facilities  not  included  in  this  study  (FAA, 
2006).  Thus,  the  centers  included  in  the  Big  6  should 
not  be  misconstrued  as  reflecting  “error  prone”  centers. 

Data  Extraction 

We  extracted  the  amount  of  time  the  radar  controller 
was  on  position  when  each  OE  occurred.  We  focused 
our  attention  on  the  radar  controller  position  because 
the  controller  in  this  position  is  primarily  responsible 
for  the  separation  of  aircraft.  For  TOP  data  when  OEs 


did  not  occur,  we  extracted  the  time  when  the  controller 
signed-on  to  position  and  the  time  when  the  controller 
signed-off.  The  difference  between  the  two  times  is  the 
number  of  minutes  a  controller  was  on  position. 

We  extracted  1,397,206  TOP  records  and  290  OE 
records.  It  should  be  noted  that  no  attempt  was  made 
to  extract  OE  severity  information.  This  decision  was 
made  for  two  reasons.  First,  the  formula  used  to  calcu¬ 
late  OE  severity  in  2006  was  different  from  that  used  in 
subsequent  years.  Thus,  we  wished  to  avoid  any  confu¬ 
sion  resulting  from  this  change.  Second,  previous  work 
failed  to  identify  any  statistically  meaningful  differences 
in  OE  severity  based  on  the  amount  of  time  a  controller 
was  on  position  (Bailey  et  ah,  2008).  Thus,  there  was  no 
empirical  evidence  that  OE  severity  was  related  to  the 
amount  of  time  a  controller  was  on  position. 


Calculations 

TOPPs  were  calculated  by  dividing  the  number  of  OEs 
that  occurred  during  a  particular  1 0-minute  time  interval 
by  the  total  number  of  controllers  who  were  signed-on 
(i.e.,  exposed  to  the  possibility  of  having  an  OE)  during 
that  time  interval.  Ten-minute  intervals  were  chosen  so 
that  direct  comparisons  could  be  made  to  previous  research 
conducted  by  Bailey,  Pounds,  and  Scarborough,  2008. 
OEm 


TOPP  = 


(Eq3) 


I  EOP, 

(  i=m 

where: 

TOPPm=  The  probability  an  OE  would  occur  within 
a  given  TOP  interval  n 

OE  =  the  number  of  OEs  for  a  given  TOP  interval  n 

TOP.  =  the  number  of  controllers  within  a  given  TOP 
interval  n 

m  =  a  given  10  minute  TOP  interval  (i.e.  “signed-on” 
time)  in  the  sequence  {1,2,3,4,5,6,7,8,9,10,11,12,13} 
corresponding  to  the  following  time  intervals  {0-10,  11- 
20,  21-30,  31-40,  41-50,  51-60,  61-70,  71-80,  81-90, 


91-100,  101-110,  111-120,  121  and  above}. 

Notice  that  the  denominator  in  Equation  3  decreased 
in  size  as  we  progressed  through  the  OE  time  intervals. 
This  is  because  the  controllers  who  were  eligible  to  commit 
the  OEs  within  a  given  time  interval  were  still  signed-on 
at  that  time.  Controllers  who  signed-off  before  a  given 
time  interval  were  not  working  traffic  and,  thus,  were 
not  eligible  to  commit  an  OE. 


11 


Table  7.  Number  of  Controllers  who  signed  off  within  a  given  time  on  position  interval 


Time  Interval  (min.) 

Interval  Range  (min.) 

Number  of  OEs 

Number  of  Controllers 

10 

0-10 

52 

133,261 

20 

10.1-20 

43 

105,675 

30 

20.1-30 

50 

123,056 

40 

30.1-40 

36 

160,631 

50 

40.1  50 

25 

193,271 

60 

50.1-60 

24 

170,684 

70 

60.1-70 

26 

138,113 

80 

70.1-80 

15 

115,063 

90 

80.1-90 

6 

96,739 

100 

90.1-100 

4 

60,521 

110 

100.1-110 

3 

33,988 

120 

110.1-120 

4 

19,332 

120.1+ 

120.1+ 

2 

46,872 

Total 

290 

1,397,206 

OEs 

Big  6  2006 


Figure  7.  Annual  number  of  OEs  distributed  by  the  amount  of  time  on  position  that 
had  lapsed  before  the  OE  occurred. 
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Time  on  Position 
Big  6  2006 


Minutes  on  Position 


Figure  8.  Annual  number  of  sign-offs  distributed  by  the  amount  of  time  on  position. 
Sign-off  refers  to  the  amount  of  time  that  lapsed  since  signing  on  to  position. 


RESULTS 

The  results  of  the  data  extraction  are  shown  in  Table 

7  in  which  the  time-on-position  and  OE  records  were 
grouped  into  13  ten-minute  categories.  The  TOP  distri¬ 
bution  of  OEs  is  presented  in  Figure  7,  and  the  distribu¬ 
tion  of  the  amount  of  time  controllers  were  on  position 
is  presented  in  Figure  8.  Notice  that  the  mode  for  the 
OE  distribution  in  Figure  7  occurs  during  the  first  time 
interval;  the  mode  for  the  TOP  distribution  of  Figure 

8  occurs  during  the  fifth  time  interval.  When  the  two 
figures  are  compared,  it  appears  that  something  other 
than  an  exposure  effect  is  causing  the  OE  peak  observed 
in  Figure  7.  Flowever,  when  theTOPP  is  calculated,  this 
proves  not  to  be  the  case. 

As  shown  inTable  8,  the  number  of  controllers  eligible 
to  commit  an  OE  is  the  total  number  who  were  still 
signed-on  to  position  (the  column  labeled  as  )  within 
a  given  time  interval  and  not  the  number  of  controllers 
who  had  signed-off .  When  the  number  of  sign-ons  is 
plotted  against  time  (see  Figure  9),  we  see  that  the  mode 
and  shape  of  the  distribution  more  closely  matches  mode 
and  shape  of  the  OE  distribution  shown  in  Figure  7. 


This  suggests  that  some  sort  of  “exposure  effect”  may  be 
operating  to  at  least  partly  account  for  the  distribution 
of  OEs  across  time  on  position. 

Table  8  further  shows  the  TOPPs  associated  with 
each  of  the  13  TOP  intervals.  When  considering  the 
Big  6  facilities  as  an  aggregate,  the  TOPPs  ranged  from 
a  low  of  0.002%  (for  the  ninth  and  tenth  intervals)  to 
a  high  of  0.006%  for  the  twelfth  interval  for  an  overall 
average  TOPP  of  0.004%,  which  is  equivalent  to  four 
OEs  out  of  every  100,000  sign-ons.  At  the  level  of  the 
individual,  this  means  that,  on  average,  a  controller  has 
a  four  in  100,000  chance  of  having  an  OE  each  time  he 
or  she  signs  on  to  position.  At  the  level  of  the  Big  6,  this 
means  that  for  every  1 00,000  position  changes,  four  OEs 
will  likely  occur.  When  plotted  across  time  (see  Figure 
10),  theTOPP  distribution  is  bimodal  with  peaks  at  the 
seventh  and  twelfth  10-minute  intervals.  However,  care 
must  be  taken  in  interpreting  the  results  due  to  the  small 
number  of  OEs  that  occurred  in  the  later  intervals.  Due 
to  the  small  number  of  OEs  (i.e.,  OEs  become  more  and 
more  rare),  the  later  probabilities  are  likely  not  to  be  as 
robust  as  those  derived  from  the  early,  and  mid-range 
time  intervals. 


13 


Table  8.  Time-on-position  probabilities  associated  with  13  time  intervals 


m 

Time 

Interval 

OEm 

TOPm 

fjOP, 

i=m 

TOPPm 

1 

10 

52 

133261 

1397206 

0.004% 

2 

20 

43 

105675 

1263945 

0.003% 

3 

30 

50 

123056 

1158270 

0.004% 

4 

40 

36 

160631 

1035214 

0.003% 

5 

50 

25 

193271 

874583 

0.003% 

6 

60 

24 

170684 

681312 

0.004% 

7 

70 

26 

138113 

510628 

0.005% 

8 

80 

15 

115063 

372515 

0.004% 

9 

90 

6 

96739 

257452 

0.002% 

10 

100 

4 

60521 

160713 

0.002% 

11 

110 

3 

33988 

100192 

0.003% 

12 

120 

4 

19332 

66204 

0.006% 

13 

120.1+ 

2 

46872 

46872 

0.004% 

Time  on  Position 
Big  6  2006 


1,400,000  n 


1,200,000 

»  1,000,000 

°  800,000 
o> 

(0  600,000 
z  400,000 
200,000 
0 

10  30  50  70  90  110  121  + 

Minutes  on  Position 

Figure  9.  Annual  number  of  sign-ons  distributed  by  the  amount  of  time  on  position.  Sign-on 
refers  to  the  number  of  controllers  who  were  still  signed  on  to  their  position  after  a  given 
amount  of  time  had  lapsed. 
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Time  on  Position  Probabilities 
Big  6  2006 


Minutes  of  Position 


Figure  10.  Time  on  position  probabilities  (TOPP)  calculated  for  each  10-minute  time 
on  position  interval. 


DISCUSSION 

The  TOPP  results  suggest  a  different  interpretation 
from  thatwhich  is  commonly  associated  with  the  analyses 
of  OE  by  TOP  The  OE  data  suggest  that  the  NAS  is 
most  vulnerable  to  OEs  occurring  early  on  position  and 
that  the  vulnerability  decreases  with  time.  In  contrast, 
the  TOPP  data  suggest  that  a  period  of  vulnerability 
may  exist  early  on  position,  but  that  the  vulnerability  is 
greatest  when  a  controller  has  worked  longer  on  posi¬ 
tion.  The  latter  interpretation  is  more  consistent  with 
the  literature  associated  with  time-on-task  fatigue,  in 
which  the  operator  experiences  greater  mental  fatigue 
the  longer  the  time  spent  on  task. 

While  it  appears  that  some  sort  of  exposure  effect  is 
responsible  for  the  greater  number  of  OEs  occurring  early 
on  position,  this  by  itself  is  not  sufficient  to  explain  the 
OE  distribution.  There  is  still  the  observed  drop  in  the 
TOPP  between  the  first  and  second  10-minute  interval. 
One  might  argue  that  the  difference  between  the  two 
(0.00001)  is  based  on  something  related  to  the  transfer 
of  position  process.  Using  this  line  of  reasoning,  one  out 
of  every  100,000  position  changes  would  produce  an 
OE  that  was  related  to  a  faulty  transfer  of  position.  This 
would  translate  to  14  OEs  during  the  first  10  minutes, 
as  compared  to  the  52  OEs  reported  in  Table  7.  Another 
way  of  saying  this  would  be  27%  of  the  OEs  occurring 


during  the  first  10  minutes  may  be  associated  with  the 
position  relief  process.  Of  course,  if  one  were  to  use  these 
numbers,  we  would  have  to  have  an  explanation,  which 
we  do  not  have,  for  a  similar  increase  in  OEs  between 
the  second  and  third  10-minute  interval.  Thus,  there 
remains  some  ambiguity  as  to  the  meaning  attached  to 
the  TOPP  distribution. 

Despite  the  above  problem,  it  is  interesting  to  specu¬ 
late  about  the  meaning  of  the  TOPP  anomaly  observed 
within  the  60  to  80-minute  intervals.  On  average,  an  en 
route  controller  spends  55  minutes  on  position.  Might 
it  be  possible  that  the  anomaly  is  an  indication  of  men¬ 
tal  fatigue  associated  with  time  on  task?  If  so,  then  how 
many  OEs  might  be  attributed  to  time-on-task  fatigue? 
If  we  use  a  TOPP  base  rate  of  0.000003,  then  anything 
above  this  might  be  attributed  to  something  unique  to 
that  time  interval.  When  this  difference  is  multiplied  by 
the  number  of  people  on  position  within  that  interval, 
the  resulting  number  is  the  number  of  anomalous  OEs. 
Using  this  process  for  the  60-  to  80-minute  intervals,  21 
out  of  the  reported  65  OEs  (35%)  appear  to  be  associated 
with  something  unique  to  that  time  interval.  Although 
we  started  with  the  assumption  that  time-on-task  fatigue 
might  be  driving  the  anomaly,  it  is  important  to  note  that 
the  available  data  are  insufficient  and  that  drawing  any 
conclusions  from  these  data  needs  to  be  substantiated 
through  other  lines  of  inquiry. 
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